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FUN3D Core Capabilities 

http://fun3dJarc.nasa.gov 




Established as a research code in late 1980’s; now supports 
numerous internal and external efforts across the speed range 

Solves 2D/3D steady and unsteady Euler and RANS equations 
on node-based mixed element grids for compressible and 
incompressible flows 

General dynamic mesh capability: any combination of 
rigid / overset / morphing grids, including 6-DOF effects 

Aeroelastic modeling using mode shapes, full FEM, CC, etc. 
Constrained / multipoint adjoint-based design and mesh adaptation 

Distributed development team using agile/extreme software 
practices including 24/7 regression, performance testing 

Capabilities fully integrated, online documentation, 
training videos, tutorials 


Conventional Adjoint-Based Design 
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• Flow field and grid adjoint equations 
derived for the time-dependent Navier- 
Stokes equations on arbitrary 
combinations of static/rigidly 
moving/deforming overset grids 
undergoing parent-child motion 

• The following terms are included in the 
Lagrangian 

• Objective function 

• Grid terms 

• Higher-order temporal terms 

• Fluxes 

• Geometric Conservation Law term 

• Overset interpolation terms 

• Initial conditions 

• Implemented by hand and verified 
using complex variables 


o is the Hadamard vector multiplication operator; see 

Nielsen, E.J. and Diskin, B., “Discrete Adjoint-Based Design for Unsteady Turbulent 
Flows on Dynamic Overset Unstructured Grids,” AIAA Journal, Vol. 51, No. 6, June 2013. 


Conventional Adjoint-Based Design 


• After linearizing the Lagrangian and solving the flow and grid adjoint 
equations, the desired sensitivities are computed as follows 
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o is the extension of the Hadamard operator to vector-matrix multiplication where the vector on the left 
multiplies each column in the matrix on the right. 


Examples 

Forward / Reverse Solutions for F-15 




• Transonic turbulent flow over 
modified F-15 configuration 

• Propulsion effects included as well 
as simulated aeroelastic 
deformations of canard/wing/h-tail 

• Objective is lift-to-drag ratio 




Reverse 

Solution 



Examples 

Forward / Reverse Solutions for Wind Turbine 




• Incompressible turbulent flow over 
NREL Phase VI wind turbine 

• Overset grids used to model 
rotating blade system 

• Objective function is based on the 
torque 


Reverse Solution 



Forward Solution 



UH-60A Blackhawk Helicopter 

Overview 



• Composite grid consists of 9,262,941 nodes / 54,642,499 tetrahedra 

• Compressible RANS: M tip = 0.64, Re tip = 7.3M, jli=0.37, a=0.0 Q 

• Blade pitch has child motion governed by collective and cyclic control inputs: 
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• Baseline value of all control inputs is zero 



Objective Function 


UH-60A Blackhawk Helicopter 

Problem Definition and Results 



Objective is to maximize C L while satisfying trim constraints over second rev: 
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• Separate adjoint solutions required for all three functions 

• 67 design variables include 64 thickness and camber variables across the blade 
planform, plus collective and cyclic control inputs up to ±7 Q 




C L 

Flow 
Solves 
(2 hrs) 

Adjoint 
Solves 
(3 hrs) 

Total Time 

Baseline 

0.023 

- 

- 

- 

Design 

0.103 

4 

4 

0.8 days 

(38,400 CPU hrs) 


Feasible region is quickly located 

Both moment constraints are satisfied within 
tolerance at the optimal solution 

Final controls: 6 C = 6.71 Q , <9 1c = 2.58 Q , <9 1s =-7.00 Q 


UH-60A Blackhawk Helicopter 

Results 





Multidisciplinary Design 

Sonic Bo om Mi tigation 

• Multidisciplinary discrete adjoint has been very successful for sonic boom 
mitigation - discrete derivatives of ground-based metrics with respect to OML 

• Many other disciplines being considered / pursued 



FUN3D FI 
Solver 
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sBOOM Adjoint 


FUN3D Adjoint 
Solver 


Sensitivity 

Analysis 




sBOOM 

(Augmented Burgers 
Equation for Propagation) 


Ground Signature, 
Loudness 




Challenges for Unsteady Problems 


• Extensive linearization and infrastructure effort, particularly for 
dynamic and overset grids 



Sheer cost - every simulation is now a time-dependent run 

• For steady flows, terms could be 
computed once and stored for efficiency 

• Unsteady flows require these 
linearizations to be recomputed at every 
time step 


Need for entire forward solution 


• Brute force it: Store to disk (big data) 

• Recompute it: Store periodically, 
recompute intermediate steps as 
needed (checkpointing) 

• Approximate it: Store periodically, 
interpolate intermediate steps as 
needed 


Chaotic flows 


FUW3D beselrte: frame 56240 





Goal of Current Work 


Compute sensitivities of infinite time 
averages for chaotic flows 


• Theory exists that states these sensitivities are well-defined and bounded 

Why does conventional approach not work? 

For chaotic flows: 

• The finite time average approaches the infinite time average 

• The sensitivity for a finite time average does not approach the sensitivity for the 
infinite time average 


l 

Chaotic shedding for 0012 
M„=0.1 Re= 10,000 a=20 l 





Adjoint solution grows 
exponentially in reverse time 
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Approach 


• Least-Squares Shadowing (LSS) method proposed by Wang and 
Blonigan 

• Key assumption is ergodicity of the simulation: long time 
averages are essentially independent of the initial conditions 

• Also assumes existence of a shadowing trajectory 

• The LSS formulation involves a linearly-constrained least squares 
optimization problem which results in a set of KKT equations 

• Preliminary LSS exploration for fluids applications 

f a 

Define the following quantities: 

Qi = Vector of conserved variables at time level i 
R t = Vector of spatial residuals at time level i 

V = Matrix of cell volumes 
t = Time 

fi = Objective function at time level i 
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LSS System 
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Adjoint solution 



((a W 

50 

51 


\S m ) 
'hj' 

h 2 

\h m j 

0 


v°y 


v j 


G = V/A t + dR/dQ 
g = cf/8Q 

h is related to time dilation 
a is a regularization parameter 


This is a globally coupled space-time 
problem , where each sub-row 
represents a time level 


Reduced LSS System 


• To determine sensitivities, we need the LSS adjoint solution 

• Use a Schur complement approach to arrive at a reduced system for 
the LSS adjoint variables: 

Writing the previous system as 
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The LSS adjoint solution can be determined from 
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• This remains a globally coupled space-time problem 

• BB r increases the fill of the matrix 

• Furthermore, the system is dense due to CCTterm 



Sensitivity Evaluation 


• To determine sensitivities, we evaluate the conventional sensitivity 
expression using the LSS adjoint solution 

• Conventional terms related to initial conditions drop out 
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Problem Definition 


Shedding NACA0012 
M„=0.1 Re=1 0,000 a=20° 



• Unstructured mesh consisting of 102,940 grid points with 100,139 prisms 
and 1 ,144 hexes in spanwise direction 

• Relatively coarse wall spacing to alleviate stiffness in LSS system 

• Laminar Navier-Stokes equations with second-order spatial discretization 

• First-order backward differencing in time for LSS simplicity 


Problem Definition 


Lift vs Time Step 



Lift vs Alpha 



Angle of Attack 






Simulation started from chaotic initial solution to improve ergodicity 
Objective is to maximize time-averaged lift over final 1 ,000 time steps 


Approach 


• Execute FUN3D flow/adjoint solvers to output data to disk for use in 
LSS: nonlinear residual vectors and Jacob ans of residual and 
objective function 

• For this tiny problem, the raw dataset is ~1 .1 TB (in-core requirement 
much larger) 

• Developed standalone LSS solver, where partitioning is performed in 
time with a single time plane per core 

• Assume the spatial discretization fits on a single core for 
simplicity 

• Global GMRES solver used with a local ILU(O) preconditioner for 
each time plane, with CC r term neglected in preconditioner 

• Execution was constrained to a subset of the cores available on each 
128 GB Haswell node to provide sufficient memory for solving the 
LSS adjoint system 

• Checked discrete consistency of LSS implementation using complex 
variables 

• This complex variable test does not provide the same rigor for LSS as 
for conventional adjoint implementations; additional verification 
approaches needed 


Residual Norm 


Solution of LSS Adjoint System 


• After ~30 minutes for I/O, solution converges 5 orders of magnitude in 
~30 mins on 2,000 cores 

• Solution remains bounded 


Convergence of LSS Adjoint System LSS Adjoint Solution for Energy Equation 
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Current Status and Future Outlook 


• Assess if (or how well) ergodicity assumption is satisfied for 
this problem 

• Evaluate quality of computed sensitivities 

• Attempt design optimization 

• How to afford extension of LSS to realistic problems? 


Thank you to the organizers 
for having us! 


